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Executive Summary 


For the past six years, California has been 
revising its preparation system for school 
and district administrators, including 
developing the California Administrator 
Performance Assessment (CalAPA), the 
first required performance assessment for 
preliminary administrator certification. In 
this study, researchers from the California 
Comprehensive Center at WestEd, the 
Principal Leadership Institute at the 
University of California at Berkeley, and 
the Regional Educational Laboratory West 
at WestEd sought to examine the initial 
impact of the CalAPA policy by identifying 
early changes (i.e., changes made through 
March 30, 2018) in administrator preparation 
programs’ practices. 


Figure 1. By March 2018, three quarters of 
respondents reported completing all or some of 
the changes to their Preliminary Administrative 
Services Credential Programs in preparation for the 
administration of the CalAPA 
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COMPLETED CHANGES 


Summary of Findings 


In March 2018, researchers conducted 
individual phone interviews with lead staff 
from 20 sampled administrator preparation 
programs to discuss the most significant 
impacts of the CalAPA policy to date in the 
areas of coursework, fieldwork, resources, 
and support. Some highlights of the findings 
from these interviews are listed below. 


Coursework 


Curriculum: Program content. 
Respondents indicated that changes, 

ranging from minor to major, have been 
made to their program’s curriculum 
as a result of the CalAPA, including 
making curriculum shifts and ensuri 


stronger alignment to the tasks in t 
CalAPA. 


Instruction. Leaders described various 
current and anticipated instructional 
changes, which are tightly interwoven 
with performance expectations 
embedded in the CalAPA. 


Course-based assignments and 
formative assessments. Respondents 
reported that program assignments 
and formative feedback were 
changing in order to better guide 

and support candidates in passing 

the CalAPA and meeting the other 
Preliminary Administrative Services 
Credential (PASC) expectations. These 
changes include altering the order of 
assignments, deepening existing tasks, 
and creating new projects. 


Fieldwork 


* Program approaches to fieldwork 
generally centered on aligning 
fieldwork to the CalAPA policy, needing 
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to provide additional professional 
learning for fieldwork supervisors, 

and linking fieldwork more tightly to 
curriculum and instructional practices. 


Resources 


Human resources. Programs 

reported the need to add staff, shift 
assignments, and create new positions; 
they described varied approaches 

to funding these additional human 
resources needed to implement the 
CalAPA policy. 


Material and equipment. 
Respondents’ comments about 
changes in material and equipment 
resources due to the implementation 
of CalAPA policy varied greatly, but 
generally centered on textbooks and 
technology. 


Time allocations. Program 
respondents all described an increase 
in staff time devoted to understanding 
and implementing the CalAPA policy. 


Support 


CTC informational activities. Many 
respondents attended a variety of 
CalAPA orientation activities offered by 
the California Commission on Teacher 
Credentialing (CTC) and reported that 
the activities provided a foundational 
understanding of the CalAPA policy 
and what was required of them. 


Increased CTC communication 

and consultation. While programs 
recognized helpful early support, 
respondents also indicated a desire for 
increased and timely communications 
from CTC about policy. 


Collaboration across programs. Many 
asked for opportunities to connect with 
colleagues across programs as they 
make changes and implement new 
approaches. 


Issues for Further Discussion 


In addition to the study’s findings, the report 
also provides commentary about key issues 
in the field of education that could affect 
administrator preparation programs and 
CalAPA implementation. 


Questions About the Assessment 


* Feedback process. Many leaders 
expressed an interest in candidates 
receiving formative and timely 
feedback on their CalAPA results and 
wondered how remediation will be 
handled for students who do not pass 
the CalAPA. 


Scope. Leaders expressed concern 
about the scope of the CalAPA. Some 
respondents worried that the CalAPA 
is more like a summative assessment 
for experienced administrators than a 
preliminary assessment for candidates 
at the start of their administrative 
careers. 


Leadership cycles. Respondents 
expressed specific concerns about 

the CalAPA leadership cycles and the 
lack of flexibility, with some suggesting 
merging cycles 1 and 2 or eliminating 
cycle 2 altogether. 


Program-Level Constraints 


* Series of changes. Respondents 
indicated that implementing the 
CalAPA policy required significant 


lifornia Administrator Performance 


icy on Preparation Programs 


changes in practice, which followed 
quickly on a series of other 
comprehensive revisions to the new 
statewide administrator credential 
system. 


Staffing structures. Respondents 
reported that their administrator 
preparation programs were constrained 
by having limited full-time faculty and 
being reliant on adjunct instructors 

and staff with limited availability for 
program development or research. 


Limited professional opportunities. 
Preliminary preparation programs 
operate in an underresourced 
professional culture with few 
professional networking and support 
options that specifically target 
administrator preparation programs 
and staff. 


Lack of research funding. The ability 
of the education field to prioritize a 
robust research agenda to study and 
guide administrator preparation is 
constrained by limited leadership 
development funds. 


Candidate-Level Constraints 


Competing commitments. All but 
one of the respondents described their 
programs as customized for full-time 
working professionals, who are mostly 
classroom teachers; this can create 
competing commitments between 
teachers’ professional responsibilities 
and their pursuit of administrator 
licensure. 


Assessment concerns. Many 
respondents expressed that the 
CalAPA’s cycles represent an 
overwhelming set of expectations 


and avery steep learning curve 

for candidates. In addition, some 
requirements of the assessment could 
create tension for the candidates with 
colleagues and supervisors. 


The Role of State Agencies in Policy 
Implementation 


Compliance and technical assistance. 
The CTC’s regulatory and disciplinary 
functions might inhibit preparation 
program leaders from revealing 
significant issues they have in 
implementing new policies such as the 
CalAPA. 


Organizational capacity. Respondents 
recognized that CTC staff have 

been responsive to the field and 

have prioritized direct support for 
preliminary preparation programs 

to the best of their organizational 
capacity. However, more and varied 
support is needed as the CalAPA, and 
other policy changes, roll out. 


Concerns About Increased 
Accountability 


Use of accountability measure. 
Respondents assumed that programs 
will be compared against each other 
and that CalAPA results will be used as 
a high-stakes accountability measure 
by the CTC, employers, and potential 
applicants to their programs. 


Narrowing of scope. Program leaders 
reported concerns that they would 
need to design preparation programs 
overly focused on the CalAPA’s 
expectations in order to ensure 

that candidates would be able to 
successfully complete the assessment. 
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Potential Challenges to the Ongoing 
Viability of Programs 


A program's finances could be thrown 
off if its enrollment numbers decline 
because its candidates cannot pass the 
CalAPA, or because candidates choose 
another program with a better pass 
rate or pursue a different preliminary 
credential pathway. 


Considerations for Next Steps 


Based on the study’s findings and the 
Issues for Further Discussion section, the 
report offers the following short-term and 
longer-term actions for consideration: 


Short-Term Considerations 


Continue to provide administrator 
preparation programs with clear and 
timely communications in various forms 
about policy requirements, timelines, 
changes, and emerging practices 
related to the CalAPA. 


Address initial questions and concerns 
about the assessment design. 


Increase efforts to educate K-12 
schools and district leadership about 
the expectations of the assessment. 


Collaborate with K-12 schools and 
district leadership to develop local 
options for meeting the demands of 
other aspects of the CalAPA policy. 


Initiate more substantive and 
collaborative support that assists 
programs in providing a variety of 

help to candidates to successfully 
complete the preparation and 
assessment process for the preliminary 
administrator credential. 


Clarify the parameters of the agency’s 
leadership capacity in convening 

and sustaining preliminary program 
networks, and support program 
leaders in forming ad hoc cohorts or 
encouraging others — professional 
organizations, foundations, or fee-for- 
service providers — to sponsor them. 


Longer-Term Considerations 


* Resolve potential policy conflicts 
between the CalAPA and the California 
Preliminary Administrative Credential 
Examination exam pathways. 


Form a collaboration of agencies, 
institutions, and stakeholders to 
advocate for funding and resources 
to strengthen the field of education 
leader preparation that now includes 
the CalAPA. 


Invest in a broader and more robust 
research agenda to inform the ongoing 
implementation of the CalAPA policy. 


Determine how much funding to 
allocate to support research in the field 
of leadership preparation during this 
period of intense policy revision and 
implementation. 


Overall, the study’s findings can be used 

by administrator preparation programs, 
policymakers, and the education field at 
large to inform possible refinements to the 
CalAPA policy; inform ways that preparation 
programs integrate and implement the 
CalAPA; and help determine how best to 
support administrator candidates through 
the preparation and assessment process. 
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Background 


For the past six years, California has been 
reviewing, revising, and strengthening its 
preparation and certification system for 
school and district administrators. This 
work included the state updating its policy 
for the Preliminary Administrative Services 
Credential (PASC), resulting in uodated 
program and professional standards, 
along with new content and performance 
expectations for candidates. 


As of summer 2018, the California 
Commission on Teacher Credentialing 
(CTC) approved 65 preliminary preparation 
programs, using the updated program 
standards. Policy changes also led to the 
development of the California Administrator 
Performance Assessment (CalAPA), the 
first required performance assessment for 
preliminary administrator certification. The 
CalAPA was piloted in 25 administrator 
preparation programs in 2017-18, field 
tested in 25 programs in 2018-19, and 

is scheduled to be fully implemented in 
2019-20. 


With the authorization of the Cal[APA 
policy,’ program leaders, instructors, field 


supervisors, and coaches in California’s 
administrator preparation programs began 
revamping various aspects of their programs 
to prepare and support candidates to pass 
the CalAPA and meet the newly established 
expectations to earn the PASC.? Even before 
the first consequential administration in 
2019-20, it appears that the CalAPA policy 
has and will continue to have significant 
impact on administrator preparation 
programs, future candidates, and the field 

at large. (For more information about the 
CalAPA, see the text box: Overview of 

the California Administrator Performance 
Assessment.) 


Purpose 


In this study, researchers from the California 
Comprehensive Center at WestEd, the 
Principal Leadership Institute at the 
University of California at Berkeley, and 

the Regional Educational Laboratory West 
at WestEd sought to examine the initial 
impact of the CalAPA policy by identifying 
early changes (i.e., changes made through 
March 30, 2018) in administrator preparation 
programs’ practices. This early study is in 
no way meant to evaluate programs or 
personnel; no program names are used in 
this report. 


Documenting reactions to a new policy that 
has not been fully implemented presents 


' At its September 2013 meeting, the CTC amended the requirements for earning a PASC to include, when 
developed, passage of an administrator performance assessment (i.e., the CalAPA). In June 2018, the CTC made 


further amendments to the CalAPA policy. 


2 In California, there are three pathways to the PASC — traditional, intern, and examination. Both the traditional 
and the intern pathways require a passing score on the CalAPA as part of the program. This study focuses on 
the traditional pathway, in which candidates must complete a Commission-approved program of specialized and 
professional preparation in administrative services and, beginning in 2019-20, achieve a passing score on the 
CalAPA. Successful completion results in the program sponsor formally recommending the candidate for the 
PASC. (Find additional details about California's PASC requirements in Appendix A.) 
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a challenge because respondents may work — administrator preparation programs, 
have strong initial reactions, which may policymakers, and the education field at 
evolve as they become more familiar with large. For instance, information from this 
the nuances of the policy. Nonetheless, study could be used to inform possible 

the findings from this study present a refinements to the CalAPA policy; prompt 
useful snapshot of the current state of additional systemic supports related to the 
preparation programs that are preparing CalAPA; inform the ways that preparation 
for consequential implementation of the programs integrate and implement 
CalAPA policy. the CalAPA moving forward; and help 
determine how best to support candidates 
through the preparation and assessment 


The study’s findings can be used in a variety 


of ways by all those connected with the process for the PASC. 


BOX 1. OVERVIEW OF THE CALIFORNIA 
ADMINISTRATOR PERFORMANCE ASSESSMENT 


In recent years, interest in authentic demonstrations of administrative ability has 
increased among administrative credentialing programs and education stakeholders. 
California is one of a small number of states that has taken action to incorporate a cap- 
stone performance assessment as a licensure requirement for the state’s Preliminary 
Administrative Services Credential (PASC). 


In September 2013, the Commission on Teacher Credentialing (CTC) revised the pro- 
gram standards and requirements for earning the preliminary credential through the 
coursework pathway. These requirements included, along with successfully completing 
an approved program and fieldwork experiences, passing an administrator perfor- 
mance assessment, when developed. In 2016, the CTC, Evaluation Research Systems/ 
Pearson, and a stakeholder Design Team began developing this new required assess- 
ment, the California Administrator Performance Assessment (CalAPA). The CalAPA is 
an intensive performance assessment carried out in three cycles over the course of a 
candidate’s preliminary administrator credential program. 


Pilot and field testing. During the winter and spring of 2017, 25 of the 65 approved 
administrator preparation programs pilot tested the CalAPA with 300 candidates. 
Results from these pilot tests informed various refinements to the CalAPA and its scor- 
ing rubrics. During the 2017-18 school year, approximately 500 preliminary candidates 
and 25 program providers participated in a field test of the CalAPA system, informing 
additional refinements to the preliminary program standards and to the CalAPA 
policies, assessment process, and rubrics. 


Non-consequential year. Prompted by comments from the field requesting more time 
to prepare for full CalAPA implementation, 2018-19 has been designated as a non-conse- 
quential year for candidate results. Non-consequential in this context means that candi- 
dates who enroll in a PASC preparation program prior to June 1, 2019, will receive a score 
on the CalAPA. However, they will not be held to meeting the CalAPA passing standard 
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as a condition for earning a Preliminary Administrative Services Credential as long as the 
candidate has completed and submitted the CalAPA for scoring and is recommended for 
the preliminary credential or certificate of eligibility by December 31, 2020. 


Full implementation. The first year of full implementation of the CalAPA begins in 
the 2019-20 academic year for all candidates enrolling in a Commission-approved 
PASC preparation program on or after June 1, 2019. As of this date, candidates who 
enroll ina PASC program must pass the CalAPA (according to the yet to be final- 
ized and adopted performance standards) as a condition for earning a Preliminary 
Administrative Services Credential. Starting June 1, 2019, candidates will be charged a 
total fee of $425.00 for the assessment. 


Assessment structure and process. The CalAPA is structured around tasks situated 
in three leadership cycles that are completed at three different periods during a 
candidate’s preliminary program. Each task focuses on the roles and responsibilities 
of today’s education leaders, using an investigate, plan, act, and reflect leadership 
sequence. Completion of each task requires that candidates either be in a school 
site-placement or have access to a school site where they can complete the work 
necessary for the CalAPA. The CalAPA approach provides an educative process that 
guides candidates to complete a leadership cycle, submit it for scoring, and receive 
assessment results. These results include a “pass” or “no pass” score, along with ana- 
lytic feedback about specific Performance Expectations. The CalAPA process allows 
programs to use assessment results from the first two leadership cycles to inform 
programs and candidates about the support needed to improve leadership practice. 


The assessment comprises the following three leadership cycles focused on school site 
level work: 


¢ Cycle 1: Planning School Improvement — Conducting data-based investigations, 
and planning and facilitating collaborative data inquiries that support equity and 
school improvement. 


¢ Cycle 2: Facilitating Professional Learning — Facilitating collaborative learning 
among a small team of teachers to improve student learning. 


¢ Cycle 3: Supporting Teacher Growth — Coaching an individual teacher to improve 
teaching and learning. 


Emphasis on multiple modalities for evidence across these three leadership cycles 
allows candidates to submit evidence in various formats: annotated video, written 
plans for implementing academic priorities, observation notes and feedback on 
teaching practice, and narrative responses and reflections about practice. Elements 
requiring video must be directed, specific, and annotated. 


Sampling that were approved by the CTC at the time 
of the study asking them to complete a 


Researchers sent three emails about the demographic survey (see Appendix B for 
CalAPA study to program leads from the the survey questions). Thirty-four programs 
65 PASC preparation programs in California answered the request to complete the survey. 
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Researchers analyzed these demographic 
survey results to create a purposive sample 
consisting of a representative mix of the 
following characteristics: 


Authorized agency type (e.g, institution 
of higher education, county office of 
education, nonprofit) 


Program size 
Geographic location 


Depth of participation in the initial 
CalAPA-related activities 


Based upon this analysis, researchers invited 
23 representative programs to participate in 
the study. Of those, 20 programs completed 
the interview with researchers (see 
Appendix C for the sample profile). 


Data Collection Process 


In March 2018, researchers conducted 
individual phone interviews with lead staff 
from sampled administrator preparation 
programs to discuss the most significant 
impacts of the CalAPA policy to date. Each 
interview lasted between approximately 
45 to 60 minutes. To assist with qualitative 
analysis, all interviews were recorded and 
transcribed verbatim. 


Researchers asked interviewees questions 
about how the CalAPA policy had changed, 
or was anticipated to change, various 
aspects of their program: 


* How the CalAPA policy changed the 
program structure of the respondent’s 
preparation program. 


How the CalAPA policy changed 
the coursework of the respondent’s 
preparation program. 
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How the CalAPA policy changed 
the fieldwork of the respondent’s 
preparation program. 


What kinds of resources the 
respondent’s program had allocated 
to meet the requirements of the new 
CalAPA policy. 


What kinds of support had helped 
inform program staff so far 


What kinds of support program staff 
anticipate will be beneficial in the 
future. 


(See Appendix D for the full list of interview 
questions.) 


Data Analysis 


Researchers conducted a detailed qualitative 
analysis of the transcribed interviews. Codes 
were generated using a combined inductive 
and deductive approach. Deductively, a set 
of initial a priori codes were created based 
on the question topics in the structured 
interview protocol. Multiple researchers 

used these topics to code and organize the 
interview transcripts, which led the research 
team to inductively generate additional 
codes for themes that emerged in the data 
from at least three separate interviews. The 
team of researchers then collaboratively 
reviewed the coded, organized text and 
identified and clarified areas of disagreement 
in the process of crafting summary findings. 
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Summary of Findings 


The findings of the study are organized 

in this section by key topics used in the 
interview protocol: Coursework, Fieldwork, 
Resources, and Support. Researchers 
identified themes within these topics and 
used respondents’ own words to describe 
completed and anticipated changes to 
program organization and practices. 


Results from the 34 programs that 
completed the initial demographic survey 
demonstrate that, two years before the first 
consequential year of the CalAPA, programs 
are already making changes. 


Coursework 


Curriculum: Program Content 


Respondents indicated that changes have 
been made to their program's curriculum as 
a result of the CalAPA (see Figure 2), ranging 
from minor to major. These changes included 
making curriculum shifts or additions and 
ensuring stronger alignment to the tasks in 
the CalAPA assessment. 


Those interviewed described changes 

in program curriculum as ranging from 
minor shifts — such as “potentially a little 
more emphasis on some alignment with 
CalAPA content, but not new content, just 
new emphasis” — to major revisions. As 
one respondent said, “This is going to be 
an extreme makeover for my program ... 
especially when the new CalAPA goes into 
effect.” Seven programs identified specific 
types of curriculum shifts or additions that 
were needed because of the new focus on 
CalAPA — content that is more intentional, 
more in-depth, and specifically addresses 
performance expectations by addressing the 


topics and skills called for in the Leadership 
Cycles. One respondent explained, “Since 
we've been part of both the pilot and the 
field testing, we've been trying to integrate 
as much as we can.... There is a focus, for 


Respondents described 
a wide range of changes in 
program curriculum from 
“a little more emphasis on 
some alignment” to “extreme 


makeover.” 


example, in our instructional leadership class 
on observation in classrooms — now the 
course is more structured.” 


Three programs indicated that focusing 
explicit attention on terminology used in 
CalAPA tasks was important in supporting 
candidates’ success “so that when 
candidates take the CalAPA, they’re not 


Figure 2. By March 2018, three quarters of 
respondents reported completing all or some of 
the changes to their Preliminary Administrative 
Services Credential Programs in preparation for the 
administration of the CalAPA 
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thinking, ‘What does that mean?’ ... And so 
that the language is not going to trip them 
up.” Another program identified adding 
content related to adult learning theory and 
also research methods for qualitative and 
quantitative data: “We're going to really have 
to be more specific and explicit about the 
research methods that we include in our 
coursework.” 


Another respondent explained that Cycle 1 
demanded that programs help students 

get “more data than were available on the 
website ... to really try to get to a root cause 
analysis.” But because 


A program leader explained the need to 
“look at a different textbook ... or some 
type of support materials for candidates to 
be able to do coaching and facilitation, not 
just talk about it.” Program respondents 
emphasized that changing instructional 
practices required analyzing CalAPA cycles 
very closely and knowing “what kind of 
skills to teach and how best to do that.” One 
respondent noted, “We had talked about it 
and so forth, but when staff actually taught 
the class, that’s when they realized, ‘Oh, I’ve 
got to prepare candidates with that skill” For 
other programs, change meant moving away 
from “having candidates 


those data are collected 

in real-life settings, 
governed by the district 
and school and not the 
preparation program, the 
director concluded that 
staff needed to work more 
closely with candidates’ 


that skill.” 


“We had talked about it 
and so forth, but when staff 
actually taught the class, that’s 
when they realized, ‘Oh, I’ve 


got to prepare candidates with 


do research papers” and 
“reading the book to 
emphasizing focusing in 
on that practice.” As one 
respondent summarized, 
“The instructional strategy 
is more project-based- 
driven and a lot of in-class 


districts to develop 
understanding of new 
expectations that, in turn, necessitate more 
complex projects and require candidates 
have access to much deeper district data. 
Still another resoondent pointed to adding 

“a coaching series and framework ... anda 
facilitation series and framework” as a means 
to fill content gaps in the previous curriculum 
of the respondent’s program. 


Instruction 


Along with curriculum content modifications, 
leaders described necessary instructional 
considerations as they prepare candidates 
for the CalAPA. Current and anticipated 
instructional changes described by 
respondents were tightly interwoven with 
performance expectations embedded in the 
CalAPA. 


practice.” 


Programs indicated that instructional 
changes centered on supporting candidates’ 
learning and their ability to demonstrate 
selected practices called for in the 
performance assessment. For example, 

one respondent described the program’s 
instructional shift as follows: “We're looking 
at our lectures and making sure that we 

are supporting ... knowing how to builda 
community of practice ... letting candidates 
be more problem solvers.” Program 
respondents noted the importance of giving 
candidates “those real-world problems 

and working through them ... and then 
coming up with some proposed solutions.” 
Another program leader reported, “We use 
all the instructional strategies ... because 

it is important for candidates to be able to 
have ... the instructional toolkit to be able 
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to coach teachers. We actually engage in 
modeling and really doing research around 
different techniques.” 


To accommodate needed instructional 
changes, another respondent suggested 
having “a little bit more time built [into 
classes] so there’s a sort of modeling of 
the data analysis, but also more time for 
[candidates] to do it with some feedback in 
class before they’re released to do it on their 
own.” For some program leaders, the shifts 
that resulted in requiring more instructional 
time have caused them to rethink the basic 
structure of their preliminary programs. 


Course-Based Assignments and 
Formative Assessments 


Respondents reported that program 
assignments and formative feedback were 
changing in order to better guide and 
support candidates in 


the program also kept its own performance 
assessment because it aligned more closely 
with its individual program outcomes. 
Another program respondent stressed 

the importance of making “adjustments 

in assignments and identifying which... 
are going to be targeted as signature 
assignments to collect the right assignment 
data to meet the state standards.” 


Three programs acknowledged that even 
with current efforts to embed formative 
assessments into coursework or fieldwork, 
they know they will need additional changes 
in the future to fully implement practices 
that not only meet program expectations, 
including the CalAPA, but also benchmark 
students’ incremental progress. Programs 
were focused on aligning assignments and 
assessments with CalAPA content, but also 
on making sure “that students are very 
comfortable going into the 


passing the CalAPA and 
meeting the other PASC 
expectations. These 
changes include altering 
the order of assignments, 
deepening existing 

tasks, and creating new 
projects. 


“We have decided next year 
not to do our end-of-year 
culminating project... 
and totally embed all three 
assessments within the fieldwork 


portion of our coursework.” 


test ... being very confident 
about completing the 

tasks associated with the 
assessment.” 


Five programs specifically 
reported redirecting 
formative assessment 


Ten program leaders described various 
alterations to existing performance tasks 
to better align with the CalAPA and their 
own program approaches. One program 
described changing a performance 
assignment based on the professional 
learning community cycle (Learning 

Cycle 2) by “placing it in a different part 
of the coursework to align better with the 
assessment process.” Another program 
now has two separate assignments for the 
improvement cycle (Leadership Cycle 2) 
because they aligned to the CalAPA tasks; 


activities into fieldwork. 

A program respondent 
explained their shift in practice: “We have 
decided next year not to do our end-of-year 
culminating project... and totally embed 

all three assessments within the fieldwork 
portion of our coursework. We are all 
rewriting our syllabi and looking at how we 
can add activities and lectures and topics 
that support those fieldwork assessments.” 
Another respondent summarized the 
program's candidate assessments within 
fieldwork as onsite coaching, direct feedback 
on practice, and shadowing the candidate. 
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Fieldwork 


Program approaches to fieldwork 
components varied but generally centered 
on fieldwork placements, alignment of 
fieldwork to the CalAPA policy, and the 

need for additional professional learning 

for fieldwork supervisors. In many cases, 
interviewees described current or anticipated 
activities that linked fieldwork more tightly to 
curriculum and instructional practices. 


Because preliminary administrator 
candidates usually enter a program already 
employed in a district, the programs 


assessment. While programs understand 
that they cannot provide students direct 
support or feedback on the CalAPA itself, 
eight programs discussed aligning fieldwork 
assignments with the cycles of the CalAPA 
so students can be more easily prepared for 
the time and rigor of the assessment. One 
program director summarized, “We'll have 
applied projects within the fieldwork that 
help support candidates with understanding 
how to do the actual assessment.” Another 
respondent explained the program's plan 
as having all of the fieldwork assignments 
together at the end of the program so 
candidates are “able to 


have little control over 
fieldwork placements. 

As one program leader 
acknowledged, “We don’t 
really place our students, 
because they just come to 
us.” Therefore, the CalAPA 


“Our candidates are where 
they are. We don’t get involved 
in placement because of the 
district. That’s a district 


employee issue.” 


take that material and 
convert it into whatever 
submission they need to 
do for CTC.” Respondents 
generally agreed that 
“compared to the typical 
field assignments, [CalAPA 


policy has generated 
few, if any, changes to 
fieldwork placements by programs. Five 
respondents directly mentioned limitations 
of their programs’ influence on fieldwork 
placements, including one who said, 
“Fieldwork placements will not change. It’s 
where the student works. That’s where the 
fieldworks are going to be done. Most of 

my students, I’d say 90 percent, if not more, 
are classroom teachers. They will do their 
field experience at the school where they 

are employed.” Another explained, “Our 
candidates are where they are. We don’t get 
involved in placement because of the district. 
That’s a district employee issue.” 


As they have little control over the fieldwork 
placements of preliminary candidates, 

when asked about fieldwork, program leads 
focused their responses on the alignment 
of fieldwork to the CalAPA policy and 


is] a lot of work. There’s no 
way around it.” 


The CalAPA, undergirded by newly adopted 
and deeper performance expectations 

(the California Administrator Performance 
Expectations) for preliminary candidates, 
shifted what fieldwork supervisors needed to 
know and be able to do to guide and support 
candidates. Ten respondents indicated that 
fieldwork supervisor training had either 
increased or changed as a result of the 
CalAPA policy. For some programs, this 
meant more strongly aligning fieldwork and 
coursework. For example, one leader stated, 
“Further alignment between fieldwork 

and what was traditionally thought of as 
coursework has been significant.” Many of 
these 10 respondents also discussed the 
need for increased time to help fieldwork 
supervisors fully understand the intent and 
requirements of the CalAPA policy. As one 
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leader summarized, “We are trying to make 
sure that the supervisors understand what 
is required of these candidates, especially 
since the CalAPA is so new.” Added to this, 
respondents indicated the need for more 
time for supervisors to calibrate themselves 
to the CalAPA rubrics and with each other. 


Resources 


Human Resources 


As programs are transitioning to the new 
CalAPA policy, they reported the need to 
add staff (both full- and part-time), shift 
assignments, and create new positions. 
Programs described varied approaches to 
funding the additional 


it’s been a significant change.... We have 
hired additional staff and are looking to hire 
more.” 


A county office provider said that they use 
two recently retired educators as consultants 
who are “absolutely essential to us being 
able to get all the feedback, and then 
collaboratively look at what we can do to 
adjust things.... Because of the importance 
of the CalAPA, we're using more of their 
time.” Another respondent indicated thinking 
about additional staffing to coordinate 

the increased number of candidates the 
respondent’s program was expecting; still 
another was considering bringing ona 

staff member to coordinate the program’s 
fieldwork assignments and supervision. 


human resources needed 
to plan for and implement 
the CalAPA policy. 


Five respondents indicated 
that the new policy had led 
to a shift in assignments 
and duties for some of 

the program supervisors 
and staff where “current 
staff are going to have 


“Without any additional 
financial resources ... our 
faculty have put in considerable 
time toward thinking about 
candidate support.... It’s 
been a significant amount of 
time allocated with very little 


compensation.” 


Another program 
described the dilemma 

of hiring adjunct teaching 
staff in order to give 
faculty the release time 
they need to revise the 
program’s courses. This 
results in the class getting 
taught by a part-time 
instructor from outside the 
program. Because adjunct 


to take on some of the 

responsibilities [related] to the CalAPA.” Staff 
whose positions were originally envisioned 
as program or assessment coordinators 
needed to spend time drafting syllabi and 
rewriting courses to align with the CalAPA. 
One program indicated that “the people 
who are doing the work are the same people 
who were doing work previously, [but] 

the work has changed and we've added 

to the work.” Five programs reported that 
implementing courses, fieldwork, and other 
supports aligned to the CalAPA required 
hiring additional staff. One such respondent 
shared, “I think from a staffing perspective 


faculty time is usually 
limited to teaching, the course “gets taught 
in isolation of the program [planning and 
coordination] and that’s not what we want to 
have happen.” 


Programs discussed different ways of 
funding additional faculty and staff needed 
to plan and implement the CalAPA policy. 
One California State University resoondent 
said, “It’s interesting — without any additional 
financial resources ... our faculty have put 

in considerable time toward thinking about 
candidate support.... It’s been a significant 
amount of time allocated with very little 
compensation.” Two programs reported 
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that they “had to do an adjustment in actual 
salary for the workload of full-time faculty” to 
allow for the significant support required by 
candidates under the CalAPA policy. Another 
explained that instructors needed to review 
textbooks to either check the alignment of 
current books to new updated curriculum 

or select new books, thus requiring them 

“to allocate funding for them to have those 
meetings.” Two programs said they provided 
small stipends for staff to help get ready 

for CalAPA’s video assignments, and two 
other programs reported “calling on our ed 
tech people” to help teach skills and “figure 
out the videos and uploading and stuff 

like that.” Three other programs indicated 


program pointed to the need to “develop 
materials such as handbooks to assist 
students in remediation so they can succeed 
in passing the tests.” 


Eleven program respondents described 
technology-related changes or additions 
prompted by the CalAPA, specifically using 
technology in courses and using technology 
for communicating, documenting 
performance tasks, and tracking candidate 
progress. Three programs reported 
purchasing collaboration software to share 
documents and post videos, as well as 
purchasing subscriptions for video examples 
of teaching to practice observation and 
feedback skills. Two 


that conversations about 
compensation or salary 
adjustments were on the 
horizon. As one of these 
program respondents 
said, “Our field staff... are 
contemplating whether 
or not they’re going to 
need to be compensated 
more than they presently 
are” because they have 
additional work in the 
field course. 


Just over half of program 
respondents described 
technology-related changes 
or additions prompted by 
the CalAPA, specifically 
using technology in courses 
and using technology for 
communicating, documenting 
performance tasks, and 


tracking candidate progress. 


respondents described 
technology-based 
learning management 
systems that help their 
programs support 
candidates in collecting 
and organizing course and 
field assignments to share 
with faculty and field 
supervisors. For example, 
one of those respondents 
said, “We... out our 


Material and Equipment 


Respondents’ comments about changes in 
material and equipment resources due to the 
implementation of CalAPA policy centered 
on textbooks and technology. 


Three programs focused on the need for new 
texts due to course changes and CalAPA 
expectations. One program leader noted, 
“We've changed all of our texts to begin to 
ensure that the skills and knowledge required 
for the CalAPA are also presented in our 
textbooks. In doing so, we had to purchase 
textbooks for all of our instructors.” Another 


fieldwork into our learning 
management system so 
that fieldwork supervisors could look at the 
work that candidates were doing and give 
them feedback.” Two programs described 
their plans to use a learning management 
system as a student-driven repository for 
candidates to archive and access their work, 
not only for program and field assignments, 
but also as a personal reference library 

for later use in their Cal[APA Learning 

Cycle submissions. 


While some of the programs stated that 
candidates were using their own laptops, 
notebooks, and smartphones for required 
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video documentation, one respondent 
indicated that the program “had to acquire 
some video cameras” to loan to candidates 
who did not have their own technology. 
Another respondent noted the need 

to revamp the university’s educational 
technology lab and “dip deeply into our 
reserves ... to better support open labs and 
video editing for performance assessments,” 
at an estimated cost of between $100,000 
to $150,000. 


Time Allocations 


Program respondents all described an 
increase in time related to implementing 
the CalAPA, especially staff time to get up 
to speed on and plan to 


this taking at least 30 to 40 hours over the 
course of the semester, per semester, for me 
and my staff... but initially it will be me.” 


In addition, the develooment of the CalAPA 
has included program staff engagement 

in the Design Committee and in piloting 
and field testing the assessment cycles. 
Several respondents indicated that their 
engagement in one or more CalAPA design 
and development activities — participating 
on the CalAPA Design Committee, working 
at a pilot site or field test site, or serving 

as an assessment scorer — helped clarify 
the CalAPA policy in practice and what 
changes their program should make to 
better prepare candidates. The biggest 
resource shift, according 


implement the necessary 
changes to accommodate 
the CalAPA policy. 


Ten programs reported 
that more time was 
needed due to new 
program administration 
activities, participation 

in the CalAPA pilot and/ 
or field test, and planning 


will be me.” 


“The biggest takeaway is 
the allocation of time to do this 
transition to CalAPA... I could 
foresee this taking at least 30 to 
40 hours over the course of the 
semester, per semester, for me 


and my staff... but initially it 


to one respondent, has 
been “staff allocation 
between all of my time and 
the three professors who 
have been teaching class 
...and then two of them 
have been helping with 

the field testing.” As one 
leader summarized, the 
CalAPA has “added 


and professional learning 

sessions. Five program respondents 
indicated that increased time was required 
for program leaders to communicate the new 
policy, program changes, and development 
processes to faculty, instructors, field staff, 
coaches, admissions clerks, and to their 
institutions, in general. The transition to 

the CalAPA, as described by one program 
leader, led to “an increase in my time, related 
to front-loading it and explaining what it’s 
about and trying to get everybody on the 
same page and get everything organized.” 
Another program leader elaborated, “The 
biggest takeaway is the allocation of time to 
do this transition to CalAPA.... | could foresee 


more responsibilities for 
administering the program as opposed to 
teaching and learning.” 


Nine program respondents described how 
program planning and staff professional 
development generated by expectations 

in the CalAPA policy led to an increase in 
work time. One person noted that “there’s 
been a lot of time spent in the development 
of what the program is going to look like ... 
a year of really planning, anticipating, and 
eventually putting in the structure that 

we will be implementing in the fall.” One 
program leader captured the thoughts of 
several others: “There will be some need for 
professional development for our full-time 
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faculty as well as our adjunct faculty on 
how to implement the CalAPA and how the 
changes are made.” A respondent reported 
that the program allocated funding for their 
instructors to meet, talk about the new 
texts, and plan together; the program also 
allocated time to meet as a whole faculty 
for professional develooment around some 
of the requirements within the CalAPA so 
they are able to support students. Another 
program emphasized that “it takes time to 
train these individuals on how to incorporate 
the CalAPA into the courses because this 

is not an email training. It needs to be face 
to face.” 


To maximize time and accommodate 


Support 


CTC Informational Activities 


Staff from the California Commission on 
Teacher Credentialing (CTC) offered a variety 
of activities in the field orienting programs to 
the new CalAPA assessment and the related 
PASC expectations. Many respondents 
attended these activities and found they 
provided a foundational understanding of 
the policy and what was required of them. 


Nine respondents acknowledged the 
recent CTC-sponsored workshops and six 
mentioned the webinars as providing vital 
basic information and allowing program 


practitioners serving 


leads to ask questions 


as instructors, another 
respondent described 
using the video- 
conferencing tool Zoom 
for learning sessions and 
updates, and spending “a 
lot more time with our staff 
so instructors know what's 
going on with the other 
folks.” A program leader 


Respondents mentioned 
CTC-sponsored activities 
provided basic information 
and allowed program leads 
to ask questions and share 
ongoing work, which helped 
build common understandings 
about the CalAPA policy and 


the assessment’s design. 


and share ongoing 

work, which in turn 
heloed build common 
understandings about the 
CalAPA policy and the 
assessment’s design. For 
example, one program 
leader appreciated 

the opportunity to “go 
through the rubrics with 


noted, “There’s been some 


our whole (program) team 


release time allotted for 

this, but very little. And | think we're going 
to have to look at more. I’m anticipating that 
there’s got to be more resources given to 
this.” 


Clearly a programs effective implementation 
of CalAPA policy hinges not only ona 

deep connection between changes in 
curriculum content, instructional practices, 
and fieldwork, but also on its capacity for 
immediate and ongoing professional learning 
for program supervisors, instructors, and 
field coordinators. 


and really think about, with 
each section of the rubric, 
where we align and where we might need to 
make changes. That was really helpful.” Four 
respondents also specifically recognized 

the value of sharing program materials, 

with San Diego State highlighted as a key 
exemplar program that offered its syllabi as 
a reference for others. “Not that you would 
replicate theirs,” one of these program 
leaders emphasized, “but it made me excited 
that there is a process that one could go 
through to think about that sort of alignment, 
and some of the instructional through lines.” 
Multiple respondents referenced other 
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helpful information sources. Three of those 
interviewed recognized the value of being a 
part of the CTC Design Committee, and three 
commented about participating in office 
hours hosted by CTC. 


Increased CTC Communication and 
Consultation 


While programs recognized helpful early 
support, the desire for increased and timely 
communications from 


to the CalAPA taxed on-site staff capacity. 
They therefore highlighted the need for more 
individualized and timely direct consultation 
for programs. As one respondent explained, 
“When it comes to physically recreating and 
redesigning your program so that it complies 
with CalAPA, we need to have more physical, 
specific (hands-on) training.... It needs to 

be focused specifically on our individual 
institutions, because mine is going to be 
different from any other institution.” Another 


CTC about policy was 

also a theme among 
respondents. In addition, 
leaders sought individual 
consultation with CTC staff 
for more specific feedback 
and ideas about their 
programs’ approach to 
these complex changes. 


“When it comes to physically 
recreating and redesigning your 
program so that it complies with 
CalAPA, we need to have more 
physical, specific (hands-on) 
training ... focused specifically 


on our individual institutions.” 


respondent suggested, 
“It would be nice to have 
someone assigned to 
campus that we could 
just bounce ideas off 

and talk with [about] the 
reasonableness of certain 
changes and if it passes 
muster, for all kinds of 
things.” 


Given the level of effort 

involved with implementing the CalAPA, 
three program leaders emphasized the 
importance of transparent timelines and 
clear, consistent communication around 
the assessment so that programs know up 
front what is going to be expected of them, 
as well as what is going to be provided as 
support. Another three leaders wanted 
more timely access to the CalAPA field test 
results, as a way to “see the areas of strength 
and the areas of growth for the cohort ... to 
be a little bit more mindful and intentional” 
in their programming as a result. Another 
leader recommended more statewide 
communication to K-12 educators to let 
prospective administrators, and the district 
leaders who employ them, know about the 
CalAPA and its new expectations. 


Four preparation program leaders also 
maintained that the substantive planning and 
administrative changes involved with the shift 


Collaboration Across Programs 


While interview respondents acknowledged 
the value of the CalAPA-related information 
provided by CTC to date, many also cited 

a need for additional practical support in 
order to move beyond policy and regulation 
toward implementation and on-the- 

ground problem solving. Many asked for 
opportunities to connect with colleagues 
across programs as they make changes and 
implement new approaches. 


One program leader said, “To just say, ‘Here 
is the policy language, but you have wiggle 
room to figure it out,’ that doesn’t helo me 
problem solve.... lt would be much more 
helpful to have small sessions with programs 
dealing with similar issues.” As a means for 
this, 1O respondents suggested creating 
peer networks where similar programs or 
institutions in the same region can share 
knowledge about their ongoing efforts 
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around CalAPA. One respondent captured 
what others described: “A lot of us are trying 
to do this on our own, you know, tapping 
into our own informal networks. It would 

be nice [to have] an opportunity to sit and 
think about what you’re doing with your 
colleagues.... Maybe a formal meeting that 
fosters informal discussion: ‘What are you 


policy already have had early effects on 
candidate and program standards, staffing, 
curriculum and instruction, and resource 
allocation. The broadened role of programs 
is an additional shift as faculty and staff 
become intermediaries who must navigate 
the competing needs of candidates, 
programs, institutions, and employers, while 


each doing? What are 

you each puzzled about? 
What do you do about 
this? What are we learning 
in this area? What are 

the obstacles?’ But not 
anything where we feel like 
we're there to learn how 


Half of respondents suggested 
creating peer networks where 
similar programs or institutions 
in the same region can share 
knowledge about their ongoing 
efforts around CalAPA. 


simultaneously striving 
to master the complexity 
of implementing the 
CalAPA policy. 


To be expected, many 
program leaders see the 
changes as dramatic 


to comply, but we’re there 
to learn from each other 
on how to build our programs.” Another 
program leader agreed, saying, “| think 
regional meetings as we roll this out for a 
year would be really helpful.... | don’t know 
how to structure that. Maybe we'd have to 
pay for it ourselves, but | would have paid to 
come.... You're face-to-face, you're talking 
with other people who are doing the same 
thing.” While a peer network was a popular 
suggestion for supporting CalAPA-aligned 
program changes, program leaders also 
indicated that organizing and initiating peer 
networks is challenged by finding resources 
to do so. 


California administrator preparation 
programs have undergone six years of 
significant change, and the crescendo 
into the consequential implementation 

of the CalAPA has yet to come. Changes 
generated by the adoption of the CalAPA 


shifts and have surfaced 
concerns and tensions. 
Despite these reactions, several leaders still 
viewed the assessment as a potential positive 
catalyst for collective staff understanding of 
the preliminary program and more effective 
candidate preparation. 


The findings in this study can perhaps be 
better understood when viewed in the 
context of other systemwide education 
issues. The findings also add useful specific 
details to the broader context that give 
practitioners a better grasp of how the 
CalAPA is impacting and interacting with 
education issues that sometimes feel 
theoretical. Accordingly, this section provides 
some related commentary about key issues 
in the field of education to consider, as 
they could affect administrator preparation 
programs and CalAPA implementation. 
Based on insights from program interviews, 
this section specifically explores questions 
about the assessment, constraints to 
administrator preparation programs, the 
role of state agencies in implementing 
policy changes, concerns about increased 
accountability, and the potential challenges 
to preliminary program viability. 
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Questions About the Assessment 


While the respondents understood that the 
CalAPA is new and still emerging, several 
voiced specific questions and suggestions 
for possible future revisions, and wondered 
how input might be addressed. 


hopeful that whoever is scoring these exams 
keeps in the back of their mind, these are 
novice administrators. They are not expert. 
They haven't even assumed an administrative 
position.” As one leader cautioned, “| worry 
that CalAPA is so complicated that we 

miss the essence of what we’re trying to 

do. It feels like the focus is so much on the 


Feedback process. Many 


how, that | worry I’ve 


leaders wondered about “I worry that CalAPA is so lost the why, and that 
the feedback process complicated that we miss the why (purpose) has got 


both for the cohort and 
individual administrator 
candidates. Respondents 
expressed that having 
general information about 
the results of the field test 


I’ve lost the why.” 


much on the how, that I worry 


essence of what we’re trying to to be driving the how 
do. It feels like the focus is so (procedure), and I'm not 


certain it is.” 


Leadership cycles. Six 
respondents expressed 


would help them better 

prepare candidates for the CalAPA. As 

one leader stated, “| want to be able to see 
where the areas of strength and the areas 

of growth for the cohort are so that | could 
take that data and be a little bit more mindful 
and intentional in how | ensure that I’m 
covering all of those standards and exposing 
[candidates] to [relevant] experiences.” In 
addition, leaders expressed an interest in 
candidates receiving formative and timely 
feedback on their own CalAPA results and 
wondered how remediation will be handled 
for students who do not pass the CalAPA. 


Scope. Leaders also expressed concern 
about the scope of the CalAPA. At least 
three respondents worried that the CalAPA 
is more like a summative assessment 

for experienced administrators than a 
preliminary assessment for candidates at 
the start of their administrative careers. 
One leader respondent voiced concern 
that candidates are having to demonstrate 
mastery of skills at the same time that 
they are learning those competencies in 
the program. As one leader said, “| am just 


specific concerns about 
the CalAPA leadership cycles. Among this 
group, two leaders suggested that cycles 1 
and 2 could be merged or run concurrently, 
as to them data analysis (cycle 1) represents 
the natural focus for any professional 
earning community (cycle 2). Another leader 
felt that cycle 1 was “repetitive” and “hard 
or students to follow,” while a different 
eader suggested that cycle 2 be eliminated 
because it isn’t “nearly as important as cycles 
land 3.” Other recommendations included 
adding an emphasis on administrative 
operations to the CalAPA, as well as allowing 
additional task flexibility (particularly in 
cycle 1) in order to afford better alignment 
with the district frameworks or priorities 
in place locally. “If there was just a little bit 
of flexibility,” one leader maintained, “we 
could make it a more powerful task, a more 
meaningful task.” 


£, 
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Constraints to Administrator 
Preparation Programs 


Program-Level Constraints 


Program context — such as the need to 
respond quickly to broad-scale policy 
changes, determine appropriate staffing 
structures, deal with limited professional 
opportunities for program faculty and staff, 
or address lack of funding for research on 
education leadership — may be fueling 
program concerns as the deadline for full 
policy implementation draws near. 


Series of changes. Respondents indicated 
that implementing the CalAPA policy 
required significant changes in practice, 


instructors and staff. Respondents described 
their part-time instructors as active 
educators who have demanding full-time 
positions as principals, district office leaders, 
and superintendents. While this staffing 
model may fulfill a program’s teaching 
needs and ensure strong connections to 
the field, it most often limits the capacity of 
faculty to be substantively involved with the 
program in other areas such as governance, 
research, or program improvement. The 
inability to support full-time, dedicated staff 
also leads to significant limitations related 
to professional develooment, collaborative 
planning time, and curriculum revision. 
Program directors often teach courses and 
serve as the singular designee for program 
administration and 


which followed quickly 

ona series of other 
comprehensive revisions 
to the new statewide 
PASC system. Additionally, 
to prepare for full 


“I’m getting whiplash with all 
these changes and adjustments. 
Don’t get me wrong. I enjoy all 


of this. It’s just overwhelming.” 


redesign responsibilities. 
They are, therefore, 
overburdened even before 
factoring in the time and 
effort needed to navigate 
CalAPA’s significant 


implementation of the 

CalAPA, programs needed to initiate early 
changes, while details about the CalAPA 
were still being field tested and finalized. 
Three program leaders voiced questions 
related to remediation, not yet knowing 
specifics, but hoping that the state will help 
provide support options for candidates who 
do not pass the CalAPA. Two respondents 
also noted that the changes generated by 
the CalAPA policy followed closely on the 
institutionalization of the recently adopted 
California Administrator Performance 
Expectations (CAPE), which inspired their 
own round of changes for many preparation 
programs. 


Staffing structures. Respondents indicated 
that their administrator preparation 
programs were constrained by having limited 
full-time faculty and being reliant on adjunct 


changes and policy shifts. 
One respondent described the difficulty of 
managing these competing demands: “I’m 
getting whiplash with all these changes and 
adjustments. Don’t get me wrong. | enjoy all 
of this. It’s just overwhelming.” 


Limited professional opportunities. 
Preliminary preparation programs 

also operate in an underresourced 
professional culture. In a state that has over 
60 preliminary preparation programs located 
in institutions of higher education, county 
offices of education, local school districts, 
charter management organizations, and 
nonprofit organizations, there are only a few 
professional networking and support options 
that specifically target preparation programs 
and staff. For example, the California 
Association of Professors of Educational 


Administration is the only California-focused 
professional organization for professors 

who teach in preparation programs and 
related doctoral programs. Several other 
organizations, such as the Association 

of California School Administrators and 

the California County Superintendents 
Educational Services Association focus on 
multiple areas within the California education 
landscape, including leadership preparation. 


Lack of research funding. The ability of the 
education field to prioritize a robust research 
agenda to study and guide administrator 
preparation is constrained by limited 
leadership develooment funds. For example, 
while the University of California (UC) system 
is intended to be a leader in research, it has 
only two administrator preparation programs 
— the Principal Leadership Institute at UC 
Berkeley and at UCLA. Neither campus has a 
research center focused on school leadership 
or leadership preparation. Other professional 
and research connections related to 
administrator preparation and development 
are offered by WestEd’s federally funded 
Regional Educational Laboratory West 

and California Comprehensive Center; 
foundations such as the Wallace Foundation, 
which invests in administrator research; and 
other entities such as the Learning Policy 
Institute. National organizations such as the 
American Educational Research Association, 
the University Council for Educational 
Administration, and the National Policy 
Board for Educational Administration are 
organizations that offer their members 
up-to-date information on issues related to 
administrator preparation and, sometimes, 
opportunities or support for related 

research. However, only a small percentage 
of individual preparation program staff are 
members of these organizations, and running 
programs with only limited full-time staff 


disqualifies the programs from institutional 
memberships. Most of these groups include 
California interests, but within a broader 
multistate or national landscape. 


Candidate-Level Constraints 


Nineteen out of 20 respondents described 
their programs as customized for full-time 
working professionals, who are mostly 
classroom teachers. While this may be 
laudable, the customization may present 
challenges. For example, in university 
systems, by offering classes on weeknights, 
weekends, and summers, programs are 
frequently considered “alternative” and 

may sometimes be marginalized in their 
respective institutions, thus restricting 
access to institutional resources and 
candidate support. Also, candidates have 
competing commitments between their 
job’s professional responsibilities and their 
pursuit of administrator licensure. They 
typically do not receive financial support, 
through employers or their programs, to 
pause their employment while engaged in 
their administrator preparation program. 
Educating working professionals also 
necessitates that fieldwork placements be 
conveniently located at the candidate’s work 
location. While proximal placement may have 
some convenience for the candidate, the 
candidate’s site may not offer the targeted 
developmental or professional opportunities 
needed for the successful completion of the 
CalAPA and preliminary credential. 


The assessment itself raised various 
candidate-related concerns expressed 

by program leaders. Many respondents 
expressed that the CalAPA’s cycles represent 
an overwhelming set of expectations and a 
very steep learning curve for programs and 
students who are generally working at least 
part time. “| could see this year, as we were 
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working with the candidates, it’s a major 
challenge for them to do what’s required in 
the [CalAPA],” one leader explained. Another 
pointed out that some of the requirements 
of the performance assessment “also impact 
their relationships with 


policies, such as the CalAPA. While CTC staff 
may wish to facilitate open dialogue about 
program challenges and uncertainties in 
knowing how to address them, program staff 
may feel some risk in being fully transparent. 


their colleagues at work, 
as well as, sometimes, 
their administrators. So it 
has a major impact all the 
way around.” For example, 
existing school and district 
labor agreements might 
not support a candidate 
who is a non-credentialed 


way around.” 


“(The CalAPA requirements] 
also impact their relationships 
with [candidates’] colleagues 
at work, as well as, sometimes, 
their administrators. So it 


has a major impact all the 


Respondents recognized 
that CTC staff have been 
responsive to the field 
and have prioritized direct 
support for preliminary 
preparation programs 

to the best of their 
organizational capacity. 
However, more and varied 


administrator observing 

a teacher and providing feedback, ora 
supervising principal might feel unwelcome 
pressure to select the candidate over 
another teacher to lead a new work group 
to accommodate the timeline required for 
CalAPA completion. 


The Role of State Agencies in 
Policy Implementation 


The CTC is an agency in the executive branch 
of the California state government, serving as 
a standards board for educator preparation 
for the public schools of California. It is 

also the official accrediting body charged 
with overseeing all of the licensing and 
credentialing of professional educators in 

the state. As with other state agencies, the 
CTC’s regulatory and disciplinary functions 
might inhibit its stakeholders — preparation 
program leaders — from revealing significant 
issues they have in implementing new 


support is needed as the 
CalAPA, and other policy 
changes, roll out and are integrated into the 
approved programs. Respondents echoed 
what the research says about best practices 
in organizational and adult learning: deep 
and ongoing implementation support is 
needed to implement changes in policy and 
practice that result in targeted improvement.? 
Program leaders and researchers also 
identified the need to develop networked 
learning communities that bring institutions 
together to “understand the true nature 

of a problem, and advance local support 
structures for improvement” (pp. 2).4 


Concerns About Increased 
Accountability 


Given that the CalAPA is the first statewide 
performance assessment required for every 
preliminary program candidate, most of the 
respondents in this study described feeling 


3 Bryk, A., Gomez, L., & Grunow, A. (2011). Getting ideas into action: building networked improvement 
communities in education. Stanford, CA: Carnegie Foundation for the Advancement of Teaching. 


4 Englebart, D. C. (2003). /mproving our ability to improve: A call for investment in a new future. IBM 


Co-Evolution Symposium. 
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increased pressure to ensure that candidates 
from their programs pass the CalAPA and 
increased pressure to meet performance 
expectations required for the preliminary 
credential. They assumed that programs 

will be compared against each other and 

the CalAPA results will 


that, but who can really question.... If you 
look at the rubric and the questions, they’re 
all about the how. The CalAPA overlooks the 
whys and the whats.” Another respondent 
simply stated, “I can’t make the program just 
a CalAPA study.” 


be used as a high-stakes 
accountability measure 
by the CTC, employers, 


“I can’t make the program 
just a CalAPA study.” 


While it is possible that 
the pressure to perform, 
concerns about program 
comparisons, and the 


and potential applicants 
to their programs. One 
respondent described it in this way: “I truly 
believe that one of the things that we have to 
be aware of is that the CalAPA has to guide 
our program because we want people to 
have some assurance that we can support 
them in being successful in taking the APA. 

If our programs are not successful with our 
participants in passing the APA, it’s going to 
bring in the question a little bit of our validity 
and our capability to be a program.” 


In addition, several respondents in this 
study reported concerns that they would 
need to design preparation programs overly 
focused on the expectations included in the 
CalAPA in order to ensure that candidates 
would be able to successfully complete the 
assessment. Examples included needing 

to align instruction and curriculum to the 
assessment language and expectations, 
place an increased emphasis on content 
assessed by the CalAPA, and eliminate other 
important but non-assessed topics. Some 
respondents described concerns regarding 
this narrowing of the curriculum in order to 
ensure success on the CalAPA. For example, 
one respondent explained, “The CalAPA 
seems to focus most on the technical versus 
theoretical. And those distinctions for me 
are really important. We want to promote 
leaders who will not just maintain the system 
we have and know how to operate within 


impulse to teach to the 
CalAPA may subside over time, these issues 
are very present and alive in programs’ 
current reality as they navigate the process 
of understanding and adapting to new 
expectations and demands. 


Potential Challenges to the 
Ongoing Viability of Programs 


The implementation of the CalAPA policy 
raises a variety of potential financial and 
structural questions and challenges for 
program directors. With few exceptions, 
administrator preparation programs function 
in an environment that requires them to 
attract sufficient candidate enrollment 

and the related tuition or fees. Therefore, a 
program's finances could be thrown out of 
balance or worse if the program’s numbers 
decline because its candidates cannot pass 
the CalAPA, or because the candidates 
choose another program with a better pass 
rate or pursue another pathway option for 
preliminary credentialing that seems easier or 
requires less of a financial investment. Several 
questions regarding financial implications 
include: What will happen to the program if 
the students don't pass the Cal[APA? What 
expenses will be incurred for a remediation 
plan? What will happen when all programs 
can be compared by pass rate? How will 
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the exam route to the credential impact my 
program once the CalAPA is consequential? 
Can my program meet student needs and 
employer hiring needs? 


Structural variation may also impact program 
viability. Currently, administrator preparation 
programs in California are housed in four 
different types of organizations: private 
universities, universities within California's 
two state university systems, county 


Respondents repeatedly identified another 
potential issue they will be facing: the 
fact that prospective administrators can 
choose to take the California Preliminary 
Administrative Credential Examination 
(CPACE), the exam-only pathway, to attain 
their preliminary credential. Given that 
CalAPA implementation is still in the early 
phase, it is difficult to predict how many 
candidates will opt to take the CPACE. 
However, if employers 


offices of education, and 
nonprofit organizations. 
The programs in these 
organizations have wide 
variance in staffing 
patterns, program length, 
and degree conferral. In 
addition, each organization 
has different admissions 


“There’s no incentive for 
them to go through the course 
of study and the learning if they 
can just more quickly take and 
pass an exam and get the same 


professional certification.” 


choose to hire a growing 
number of administrators 
who have forgone 
preliminary programs 
because of the availability 
of CPACE, program- 
provided administrator 
preparation could 


policies and program 

delivery methods. This variation ultimately 
has an impact on how programs are able to 
address the requirements of the CalAPA. 
Accordingly, as programs grapple with 
implementation of the Cal[APA, some system 
questions will likely be addressed, such as: 
How will programs balance issues of cost, 
time, and successful completion of the 
CalAPA? Are some organization types more 
able to flexibly respond to the implications 
of the CalAPA policy? With the likelihood 

of increased costs for candidates and/or 
programs, will enrollments shift among the 
four types of organizations? 


diminish significantly 

due to low enrollment. 

As one respondent noted, “Once [CalAPA] 
becomes consequential ... many candidates 
are going to say, ‘No, maybe I'll just study 
for that CPACE, even though | know that 
the pass rate isn’t very high.’ They'll take 
their chances on doing that, and will 

forgo any of the preparation programs in 
institutions and county offices.” Another 
respondent concluded, “There’s no incentive 
for them to go through the course of study 
and the learning if they can just more 
quickly take and pass an exam and get 

the same professional certification. And 
now we basically have taken the exam and 
embedded it in the one-year program by 
adding the CalAPA.” 
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leaders in forming ad hoc cohorts or 
encouraging others — professional 
organizations, foundations, or fee-for- 
service providers — to sponsor them. 


Based on the study’s findings, and the 
systemic challenges explored in the Issues 
for Further Discussion section, there are 
several short-term and longer-term actions 
that the CTC, along with policymakers, other 
state-level agencies, and educators in the 
field may want to consider carrying out. 


Longer-Term Considerations 


Resolve potential policy conflicts 
between the CalAPA and the CPACE 
exam pathways. 


Form a collaboration of agencies, 
institutions, and stakeholders to 
advocate for funding and resources 
to strengthen the field of education 
leader preparation that now includes 
the CalAPA. 


Short-Term Considerations 


Continue to provide administrator 
preparation programs with clear and 
timely communications in various forms 
about policy requirements, timelines, 
changes, and emerging practices 
related to the CalAPA. 


Invest in a broader and more robust 
research agenda to inform the ongoing 
implementation of the CalAPA policy. 


Address initial questions and concerns ; ; 
Determine how much funding to 


about the assessment design. 


Increase efforts to educate K-12 
schools and district leadership about 


allocate to support research in the field 
of leadership preparation during this 
period of intense policy revision and 


the expectations of the assessment. implementation. 


Collaborate with K-12 schools and 
district leadership to develop local 
options for meeting the demands of 
other aspects of the CalAPA policy. 


Potential Areas for Further Study 


Key questions to explore in a longitudinal 
study of the impact of the CalAPA policy and 


Initiate more substantive and assessment could include: 


collaborative support that assists 
programs in providing a variety of 
help to candidates to successfully 
complete the preparation and 
assessment process for the preliminary 
administrator credential. 


Clarify the parameters of the agency’s 
leadership capacity in convening 

and sustaining preliminary program 
networks and support program 


How do preparation programs continue 
to respond to the CalAPA policy with 
further implementation? 


How does the field perceive the value 
of the CalAPA policy? 


What effect has the CalAPA policy had 
on the numbers of candidates enrolled 
in the preparation program routes and 
the exam route? 
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* How does the CalAPA policy impact 
programs’ understanding and use of 
performance assessment over time? 


How does the CalAPA policy intersect 
with other new administrative 
credentialing policy components 

(e.g., induction)? 


Additional specific studies can also be 
conducted to continue investigating the 
different dimensions of change that were 
surveyed in this early descriptive study. 
Potential research questions for these studies 
include: 


* How does the CalAPA policy impact 
preparation program structures and 
design? 


How does the CalAPA policy impact 
the curriculum and instruction of 
preparation programs? 


What do school and district employers 
report about the job-readiness level of 
program candidates who have passed 
the CalAPA as a part of licensure? 


How do the programs allocate 
resources to fulfill the CalAPA policy 
requirements over time? 


What types of effective program 
supports are developed to support full 
CalAPA policy implementation? How 
are they organized? Who do they help? 


How do the preparation programs’ 
approach to or understanding of the 
CalAPA policy shift over time? 


How do programs cope with the large 
number of changes in the field? 


This study illustrates that the CalAPA 

policy is compelling wide-scale change 

in preliminary administrator preparation 
programs across California, even before the 
assessment is finalized and consequential for 
candidate licensure. Capturing early effects 
of a policy is difficult, as 


our program. So in that sense, | think it’s a 
positive.” 


The new policy aims to improve the 
preparation of preliminary candidates so, 
once eligible, they are ready to step into 
administrative positions. One respondent 
described, “After our students finished a 
section of the CalAPA in their fieldwork and 
in their coursework, they 


implementation activities 
are only just beginning 
and programs’ reactions to 
the policy are evolving as 
the policy itself evolves. In 
addition to the expected 
prioritization of attention 
on new expectations and 
corresponding shifts in 
practice, revisions to the 
CalAPA are occurring 
simultaneously with 


“I just think it’s a good 
opportunity for our program. 
We have had a lot of turnover 


faculty-wise over the last seven 


or eight years, and I think this is 
... causing our faculty to have a 
better common understanding 
of our program. So in that 


sense, I think it’s a positive.” 


found it really helpful. 
They learned a lot from 
some of the cycles, and 
one of them was the 
coaching cycle. Although 
it was really difficult for 
them to implement, they 
really appreciated it. They 
learned a lot from it.” 
Another program leader 
said, “Personally, | think 


planning and start up. 

In addition, several policy details, such as 
the set point for passing scores or types of 
support for remediation, are not yet fully 
determined or approved by the Commission. 


Though preliminary program leaders still 
have unanswered questions about the details 
of the policy, how the CalAPA process will 
work when fully designed, and other related 
concerns, several of those interviewed 
acknowledged that changes required in 

the planning and implementation of the 
CalAPA have already shown value for staff 
and candidates. As one leader expressed, 

“| just think it’s a good opportunity for our 
program. We have had a lot of turnover 
faculty-wise over the last seven or eight 
years, and | think this is ... causing our faculty 
to have a better common understanding of 


CalAPA is all good stuff, 
and | think it’s going to make our candidates 
better administrators.” 


Conversations among educators and 
policymakers need to continue and expand 
so that programs can learn from each other 
and mature their approaches in providing 
high-quality programs for the range of 
diverse candidates seeking a license to lead 
California’s schools and districts. Along with 
carrying out a focused and funded research 
agenda to inform preliminary preparation 
for administrators, ongoing attention to 

the effects of the CalAPA will help the field 
understand whether and how California’s 
first required administrator performance 
assessment is, in fact, strengthening the 
readiness of early administrators to lead 
schools and districts to improve education 
for a// students. 
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Appendix A. 
Requirements for 
the Preliminary 
Administrative 


Services Credential 


California has a two-tier credential structure. 
A five-year Preliminary Administrative 
Services Credential (PASC) is the first 
credential issued after an individual meets 
basic credential requirements. A clear 
credential is issued when all credential 
requirements have been completed. 


For the five-year PASC for individuals 
prepared in California, individuals must 
satisfy all the following requirements: 


1. Possess a valid prerequisite credential 
2. Complete one of the following: 


a. ACommission-approved program 
of specialized and professional 
preparation in administrative services 
and, beginning in 2019-20, a passing 
score on the California Administrator 
Performance Assessment 
(CalAPA), resulting in the formal 
recommendation of the program 
sponsor 


A one-year Commission-approved 
administrative services intern 
program consisting of supervised 
in-service training resulting in the 
formal recommendation by the 
California college or university where 
the program was completed, which 
beginning in 2019-20 will also require 
a passing score on the CalAPA 


c. Achieve a passing score on the 
California Preliminary Administrative 
Credential Examination (CPACE), 
July 2015 revised version, 
administered by Evaluation Systems, 
Pearson 


3. Satisfy the basic skills requirement 


4. Complete five years of full-time 
experience 


5. Verify employment in an administrative 
position. (California Commission on 
Teacher Credentialing, 2017) 


Source: California Commission on Teacher 
Credentialing. (2017). Administrative Services 
Credential for Individuals Prepared in 
California. Sacramento, CA: Author. 
httos://www.ctc.ca.gov/docs 


default-source/leaflets/cl574c. 


odf?sfvrsn=8aa3c37f_ 8 
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Appendix B. Demographic Survey Questions 


IHE 
County Office 
Non-Profit 


District 


Other 


Intern ASC 
Preliminary ASC 


O Intern and Preliminary ASC 


Fewer than 25 
25-50 

51-100 

Over 100 per year 


O Urban 
Suburban 


Rural 


North Coast, including Del Norte, Humboldt, Mendocino, Lake, and Sonoma counties 


Northeastern, including Siskiyou, Modoc, Trinity, Shasta, Lassen, Tehama, Plumas, 
Glenn, and Butte counties 


Capital, including Colusa, Yuba, Sierra, Sutter, Nevada, Placer, Yolo, El Dorado, 
Sacramento, and Alpine counties 
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Bay Area, including Marin, Napa, Solano, San Francisco, Alameda, Contra Costa, and 
San Mateo counties 


South Bay, including Monterey, San Benito, Santa Clara, and Santa Cruz counties 


Delta Sierra, including San Joaquin, Amador, Calaveras, Stanislaus, and Tuolumne 
counties 


Central Valley, including Fresno, Merced, Mariposa, Madera, Kings, and Tulare 
counties 


Costa Del Sur, including San Luis Obispo, Kern, Santa Barbara, and Ventura counties 


RIMS, including Mono, Inyo, San Bernardino, and Riverside counties 
Los Angeles 


Southern, including Orange, San Diego, and Imperial counties 


Statewide 


CalAPA Pilot Test site 
CalAPA Pilot Test scorer 
CalAPA Field Test site 
CalAPA Field Test scorer 


CalAPA Design Team member 


CalAPA workshop (any- Madera, San Diego, Santa Clarita) 


Mostly in-person 
Mostly online 


O Hybrid of in-person and online 


Completed implementation of all changes 
Implemented some changes or in the process of implementing changes 
Planning some changes 


Initiated discussions about potential changes 


No discussions so far 
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Appendix C. Study Sample Characteristics 


5 county offices of education 4 private universities 


8 California State University 
campuses 


1 University of California campus 


2 other types of agencies 


10 programs offered intern and 10 programs offered preliminary 
preliminary only 


4 programs have less than 25 5 programs have 51-100 


6 programs have 25-50 5 programs have greater than 100 


Tl mostly in-person 6 hybrid of in-person and online 


3 mostly online 


6 urban 3 urban/suburban 
4 suburban 6 urban/suburban/rural 


lrural 


1 North Coast 
2 Northeastern 
1 Capital 

4 Bay Area 

1 South Bay 


1 Delta Sierra 


2 Central Valley 
3 Costa Del Sur 
3 RIMs 

3 Los Angeles 
3 Southern 


3 Statewide 
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11 programs served as CalAPA pilot ¢ 4 programs had staff serving as 
test sites CalAPA design team members 


6 programs had staff serving as 18 programs had staff who attended 
CalAPA pilot test scorers CalAPA workshops 


13 programs served as CalAPA field 
test sites 


3 programs completed 4 programs were planning some 
implementation of all changes changes 


12 programs implemented some 1 program initiated discussions 
changes or were in the process of about potential changes 
implementing changes 


* Some programs serve more than one geographic area, so total is more than total number in 
the sample. 
** Many programs participated in multiple activities, so total is more than total number in the sample. 


Note: Geographic regions are based on the following California County Superintendents Educational 
Services Association regions (http://ccsesa.org/members/region-map/): 


¢ North Coast: Del Norte, Humboldt, ¢ Delta Sierra: San Joaquin, Amador, 
Mendocino, Lake, and Sonoma counties Calaveras, Stanislaus, and Tuolumne 


Northeastern: Siskiyou, Modoc, Trinity, counties 
Shasta, Lassen, Tehama, Plumas, Glenn, Central Valley: Fresno, Merced, Mariposa, 
and Butte counties Madera, Kings, and Tulare counties 


Capital: Colusa, Yuba, Sierra, Sutter, Costa Del Sur: San Luis Obispo, Kern, 
Nevada, Placer, Yolo, El Dorado, Santa Barbara, and Ventura counties 


Sacramento, and Alpine counties RIMS: Mono, Inyo, San Bernardino, and 


Bay Area: Marin, Napa, Solano, Riverside counties 
San Francisco, Alameda, Contra Costa, Los Angeles 


ti 2 
euro Mabe counties Southern. Orange, San Diego, and 


South Bay: Monterey, San Benito, Imperial counties 


Santa Clara, and Santa Cruz counties Statewide 


(The following interview protocol was used 
to conduct interviews with the study’s 
participants in March 2018.) 


In this study, researchers from the California 
Comprehensive Center at WestEd and the 
University of California at Berkeley seek 

to identify the initial impact of the CalAPA 
policy on preparation programs, as well as 
potential future implications to inform the 
ongoing policy implementation. | want to 

be explicit that this study is about CalAPA 
policy, that is any of the potential implications 
of the first required performance assessment 
for administrator licensure in California, 

not exclusively the assessment itself. With 

a better understanding of the early and 
potential program changes as a result of the 
CalAPA policy, all those connected to this 
work can determine how best to support 
candidates through this process. 


While the final report will be shared with 
the California Commission on Teacher 
Credentialing, this report is not soonsored by 


the CTC nor will data be shared with the CTC. 


Strict confidentiality will be maintained 
with this report. Interview data will be 
analyzed for cross-cutting patterns and 
themes rather than analyzed separately. 
No program names will be used in the 
publication. This study is inno way meant 
to evaluate programs or personnel. Rather 
it is intended to inform the ongoing 
implementation of the policy so that all 
those connected to this work can determine 
how best to support candidates through 
this process. 


To assist in analysis, all interviews will be 
recorded and transcribed. Do you consent 
to be interviewed about the Initial Impact of 
the CalAPA Policy? Do you give permission 
to audio record and take notes during the 
interview so that we can accurately record 
the information you provide given that it will 
be used for transcription purposes only? 


When the research is completed, we may 
save the recordings and notes for use in 
future research done by myself or others. We 
will retain these records for up to five years 
after the study is over. 


Do you have any questions or comments 
before we begin? 


As you saw from the overview of the 
questions | sent, we will be asking you about 
4 main areas of potential changes to your 
program — structure, coursework, fieldwork, 
and resources. We will conclude with some 
questions about support. For each of the 

4 main sections, | will be asking you about 
current and anticipated changes. Each of 
these sections will have both open-ended 
and multiple-choice questions where | will 
ask you to rate the level of change that has 
happened so far with your program. The 
choices will be: 


O = No change 

1= Little change 

2 = Moderate change 
3 = Significant change 


4 = Extreme change 


Program Structure 


1. 


We are interested in understanding how the CalAPA policy is changing the program 
structure of your preparation program. We will begin by discussing current changes 
and then conclude with a question about anticipated changes. 


a. Have you a/ready made changes in the area of program structure? If so, please 
describe one or two examples. 


b. To what degree has staff a/ready made changes to the program structure 
(e.g., delivery method, length, or composition of the programs)? 


i) Please rate the level of change: 
O = No change 
1= Little change 
2 = Moderate change 
3 = Significant change 
4 = Extreme change 


c. To what degree has staffing structure a/ready been affected? 
i) Please rate the level of change: 
O = No change 
1= Little change 
2 = Moderate change 
3 = Significant change 
4 = Extreme change 


d. To what degree have hiring practices a/ready been affected? 


i) Please rate the level of change: 
O = No change 
1= Little change 
2 = Moderate change 
3 = Significant change 
4 = Extreme change 


e. Are there any anticipated changes to program structure as a result of the policy? 
Please describe. 


Il. Coursework 
2. We are interested in understanding how the CalAPA policy is changing the 
coursework of your preparation program. We will begin by discussing current 
changes and then conclude with a question about anticipated changes. 


a. Have you already made changes to program curriculum, instruction, and/or 
assessment as a result of the policy? If so, please describe one or two examples. 


. To what degree has content or curriculum a/ready been changed? 
i) Please rate the level of change: 
O = No change 
1= Little change 
2 = Moderate change 
3 = Significant change 
4 = Extreme change 


c. To what degree have instructional strategies a/ready been changed? 
i) Please rate the level of change: 
O = No change 
1= Little change 
2 = Moderate change 
3 = Significant change 
4 = Extreme change 


d. To what degree have the course-based assignments and/or assessments a/ready 
been changed? 


i) Please rate the level of change: 
O = No change 
1= Little change 
2 = Moderate change 
3 = Significant change 
4 = Extreme change 


e. Are there any anticipated changes to coursework as a result of the CalAPA policy? 
Please describe. 
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Il. Fieldwork 


IV. Resources 
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d. To what degree has the new CalAPA policy a/ready changed time allocations? 
Please rate the level of change: 
O = No change 
1= Little change 
2 = Moderate change 
3 = Significant change 
4 = Extreme change 


e. Are there any anticipated changes to program resources? If so, please describe. 


. What kinds of CalAPA policy support have helped inform program staff so far? 


. What kinds of CalAPA policy support do program staff anticipate will be helpful with 
implementation in the future? 


. Is there anything else you would like to tell us about the impact or potential impact 
of the CalAPA policy? 


